feat(runtime): use micromamba instead of mamba and fix build issue #4154

xingyaoww · 2024-10-01T21:08:33Z

Include this change in the Release Notes. If checked, you must provide an end-user friendly description for your change below

End-user friendly description of the problem this fixes or functionality that this introduces

Improve the docker runtime build speed & fix bugs in build

Give a summary of what the PR does, explaining any non-trivial design decisions

micromamba is a fully statically-linked, self-contained, executable. This means that the base environment is completely empty. The configuration for micromamba is slightly different, namely all environments and cache will be created by default under the MAMBA_ROOT_PREFIX environment variable. There is also no pre-configured .condarc/.mambarc shipped with micromamba (they are however still read if present).

This means micromamba is faster to install than miniforge/mamba, which will help in evaluation scenario where you may need to build thousands of containers..

This PR:

Switch from miniforge3 to micromamba
Use the openhands environment, instead of the base environment - because No other packages should be installed to base and we've been using this wrong for a while - I believe this is the cause of [Bug]: Runtime failed to build due to Mamba Error #4153
Force re-build on E2E test & browser gym runtime tests

Link of any specific issues this addresses

May fix #4153

Should also fix #4164

Some explanation of how this can fix #4153 #4164

slack message link

You build an image from scratch with mamba x.x.8 -> oh_v0.9.7_XXX
You skip init and rebuild a new image using FROM oh_v0.9.7_XXX and you try to install playwright
while installing playwright, mamba's dependency resolver decide that you should downgrade mamba x.x.8 to x.x.7 in the base environment
bingo -- mamba is now broken in this issue [Bug]: Runtime failed to build due to Mamba Error #4153.

The root issue is that we install useful stuff (playwright) in the base environment (where core mamba/conda reside), which is discouraged: https://mamba.readthedocs.io/en/latest/user_guide/troubleshooting.html#no-other-packages-should-be-installed-to-base

tobitege · 2024-10-01T21:21:16Z

in the docker script:

RUN mkdir -p /openhands/micromamba &&     curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj -C /openhands/micromamba

I think the tar is creating a different folder structure below micromamba?
Maybe this works?

RUN curl -Ls https://micro.mamba.pm/api/micromamba/linux-64/latest | tar -xvj -C /tmp && \
    mv /tmp/micromamba /openhands/

RUN /openhands/micromamba/bin/micromamba create -n openhands python=3.11 -y
RUN /openhands/micromamba/bin/micromamba install -n openhands -c conda-forge poetry -y

xingyaoww · 2024-10-01T21:24:31Z

You are probably right! Though the weird thing is i can build the exact dockerfile locally, as well as on the remote runtime 😓. I confirmed that this PR allow the previously fail image to successfully build and run

openhands/runtime/utils/runtime_templates/Dockerfile.j2

enyst · 2024-10-01T22:23:39Z

Just to note for the record: when we have switched from miniconda to miniforge3, part of the discussion there had a licensing reason, because anaconda's own licensing is messy. I made a quick check around micromamba from the licensing point of view and it looks good to me. 👍

(it's similar to miniforge3, BSD 3-clause licensed software, scripts, default conda-forge channel.)

xingyaoww · 2024-10-01T22:29:36Z

Set CONDA_FORGE_YES to yes seems to also include Anaconda's commercial source when I'm testing the RemoteRuntime:

 ---> be5325871eaf
Step 8/12 : RUN /openhands/micromamba/bin/micromamba create -n openhands python=3.11 -y &&     /openhands/micromamba/bin/micromamba install -n openhands conda-forge::poetry -y
 ---> Running in ee1c0d20abcf
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/
warning  libmamba 'repo.anaconda.com', a commercial channel hosted by Anaconda.com, is used.
    
warning  libmamba Please make sure you understand Anaconda Terms of Services.
    
warning  libmamba See: https://legal.anaconda.com/policies/en/

error    libmamba Could not lock non-existing path '/root/.mamba/pkgs'

Try set it back to no to see if it work (i think conda-forge::poetry which be equivalent to -c conda-forge which might work?)

enyst · 2024-10-01T22:32:13Z

Set CONDA_FORGE_YES to yes seems to also include Anaconda's commercial source when I'm testing the RemoteRuntime:

That's strange! I think the install script does this?

# Initializing conda-forge
case "$CONDA_FORGE_YES" in
  y|Y|yes)
    "${BIN_FOLDER}/micromamba" config append channels conda-forge
    "${BIN_FOLDER}/micromamba" config append channels nodefaults
    "${BIN_FOLDER}/micromamba" config set channel_priority strict
    ;;
esac

This reverts commit cdabea0.

This reverts commit e04c84e.

This reverts commit 6330b52.

tobitege

LGTM, awaiting the tests go through.
Thanks for the hard work! 🙏

use micromamba instead of mamba for effeciency

ffa94d9

xingyaoww mentioned this pull request Oct 1, 2024

[Bug]: Runtime failed to build due to Mamba Error #4153

Closed

2 tasks

(debug) ls

dfbc062

xingyaoww added 3 commits October 1, 2024 21:31

(debug) try directly using install script (likely more robust long term)

33c5494

fix test runtime

374e1ac

fix test runtime build

cf13e56

enyst reviewed Oct 1, 2024

View reviewed changes

openhands/runtime/utils/runtime_templates/Dockerfile.j2 Outdated Show resolved Hide resolved

conda forge yes

156a679

tweak conda forge to no

a95245c

xingyaoww and others added 16 commits October 1, 2024 22:46

tweak to install py together with poetry & set forge_yes to true again

ef76b1e

re-order env var

d883679

always remove commercial channel 'defaults'

23d755d

try log stderrr

dc8db3f

remove python unbufferred to fix remote runtime

c68efe3

add more printing

f10f98a

fix print

7d0fa1d

fix runtime build tests

08b042e

fix test

b778f44

update CalledProcessError

6999e90

add test rerun

334389a

attempt to fix ci by set n=1

cdabea0

Revert "attempt to fix ci by set n=1"

f959421

This reverts commit cdabea0.

force rebuild for browsergym test

d732d3b

try print stderr

e04c84e

force rebuild for dummy agent

3fbff15

xingyaoww added 8 commits October 2, 2024 18:34

include stdout/stderr for failed test

bc39c1d

always disable ps1

33c5238

tweak pytests

6330b52

install curl too

8a9033d

fix runtime build tests

811ba82

Revert "try print stderr"

61c6ec9

This reverts commit e04c84e.

Revert "tweak pytests"

42157a9

This reverts commit 6330b52.

Merge branch 'main' into xw/micromamba

099aec9

xingyaoww changed the title ~~feat(runtime): use micromamba instead of mamba for effeciency~~ feat(runtime): use micromamba instead of mamba and fix build issue Oct 2, 2024

xingyaoww marked this pull request as ready for review October 2, 2024 20:56

tobitege approved these changes Oct 2, 2024

View reviewed changes

xingyaoww enabled auto-merge (squash) October 2, 2024 21:13

xingyaoww merged commit e81c559 into main Oct 2, 2024

xingyaoww deleted the xw/micromamba branch October 2, 2024 21:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(runtime): use micromamba instead of mamba and fix build issue #4154

feat(runtime): use micromamba instead of mamba and fix build issue #4154

xingyaoww commented Oct 1, 2024 •

edited

Loading

tobitege commented Oct 1, 2024 •

edited

Loading

xingyaoww commented Oct 1, 2024

enyst commented Oct 1, 2024

xingyaoww commented Oct 1, 2024

enyst commented Oct 1, 2024

tobitege left a comment

feat(runtime): use micromamba instead of mamba and fix build issue #4154

feat(runtime): use micromamba instead of mamba and fix build issue #4154

Conversation

xingyaoww commented Oct 1, 2024 • edited Loading

tobitege commented Oct 1, 2024 • edited Loading

xingyaoww commented Oct 1, 2024

enyst commented Oct 1, 2024

xingyaoww commented Oct 1, 2024

enyst commented Oct 1, 2024

tobitege left a comment

Choose a reason for hiding this comment

xingyaoww commented Oct 1, 2024 •

edited

Loading

tobitege commented Oct 1, 2024 •

edited

Loading